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20.  Continued 

The  Army  Training  Board  (ATB)  and  the  Infantry  School  (USAIS)  selected 
114  CMF  tasks  for  this  test.  All  training  and  data  collection  were  conducted 
by  officers  and  NCOs  from  TRADOC  and  USAIS.  Data  analysis  was  performed  by 
MGA  staff  members. 

Sixty-eight  soldiers  were  selected  at  random  from  two  new  manning  compa¬ 
nies  to  take  part  in  this  test.  Participants  received  standard  P01  task  in¬ 
formation  and  demonstration,  and  were  then  tested  on  the  task.  First  trial 
data  were  recorded  by  the  task  force  and  transmitted  to  MGA  for  analysis. 

MGA  produced  the  following  estimates  based  on  this  first  trial  data: 
training  criterion;  the  minimum  End-of -Course  scores;  the  retention  interval 
in  weeks  for  a  decay  to  60"  correct  performance. 

The  soldiers  were  trained  to  criterion,  returned  to  the  POI ,  and  then 
retested  on  the  tasks  the  day  before  POI  refresher  training. 

ATB  selected  40  tasks  for  in-unit  testing.  Eighteen  tasks  were  tested 
at  Fort  Campbell  after  a  nineteen  week  interval  and  twenty-two  tasks  were 
tested  at  Fort  Lewis  after  a  twenty-one  week  interval. 

The  major  findings  of  this  operational  test  were: 

•  The  average  MGA  end-of-course  estimates  were  within  two  percentage 
points  of  the  average  observed  end-of-course  scores  for  tasks 
trained  to  criterion. 

•  MGA  software  was  calibrated  to  provide  retention  estimates  at  two 
levels  of  confidence.  At  the  99%  level  of  confidence  it  was  ex¬ 
pected  that  99%  of  the  observed  task  performance  data  would  exceed 
MGA  estimates.  Thirty-eight  out  of  forty,  or  95%  of  the  tasks  met 
this  expectation.  At  the  80%  confidence  level  it  was  expected  that 
80%  of  the  observed  task  performance  data  would  exceed  the  MGA 
estimates.  Thirty-five  out  of  forty,  or  87.5%  of  the  tasks  met  this 
expectation. 

•  For  these  first  trial  data,  MGA  software  predicts  that  training  to 
criterion  one  GO  should  be  sufficient  exposure  to  sustain  skill. 
Ninety-one  percent  (91%)  of  the  Fort  Lewis  soldiers  scored  two 
consecutive  perfect  trials  after  meeting  criterion  one  GO.  These 
soldiers  took  72%  fewer  sustainment  training  trials  to  reach 
criterion  that  they  had  required  to  learn  the  same  tasks  in  the 
training  base. 
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BRIEF 


Objective; 

The  objective  of  this  effort  was  to  perform  an  operational  test  of  the  capability 
of  MGA  Skill  Training  and  Retention  (STAR  )  software  to  estimate  training  and 
retention  outcomes  based  on  a  small  sample  of  first  trial  data.  This  objective 
was  threefold:  1)  To  test  the  ability  of  STAR®  software  to  estimate  soldier 
proficiency  at  end-of-course;  2)  to  test  the  ability  of  STAR®  to  forecast  soldier 
skill  decay  over  time;  and,  3)  to  determine  the  frequency  of  training  required  to 
sustain  soldier  skills. 

Procedures: 

The  Army  Training  Board  (ATB)  and  the  Infantry  School  (USAIS)  jointly  selected 
114  CMF  11  and  soldier's  common  tasks  for  this  test.  MGA  staff  produced 
performance  measures  that  were  reviewed  by  USAIS  staff  and  used  for  testing 
and  training  the  tasks.  All  training  and  data  collection  were  conducted  by  a  task 
force  of  officers  and  NCOs  from  TRADOC  and  USAIS.  Data  analysis  was 
performed  by  MGA  staff  members. 

Soldiers  from  platoons  of  two  new  manning  training  companies  participated  in 
this  effort.  Participants  received  standard  POI  task  information  and  demon¬ 
stration  and  were  then  tested  on  the  task.  First  trial  data  were  recorded  by  the 
task  force  and  transmitted  to  MGA  for  analysis. 

STAR"-'  analysis  produced  the  following  estimates  based  on  this  first  trial  task 


5SE  training  criterion:  the  number  of  successively  correct  (GO) 
trials  the  soldiers  were  required  to  perform  in  training  so  that 
30%  had  stopped  making  errors.  MJSAI5  selected  this  criterion.) 


STAR12,  estimate  of  the  minimum  End-of-Course  performance. 


© 

•  STAR  estimate  of  the  retention  interval  in  weeks  for  a  decav 
to  60%  correct  performance. 

The  soldiers  were  trained  to  criterion,  returned  to  the  POI  and  then  retested  on 
the  tasks  the  day  before  POI  refresher  training. 

ATB  selected  40  tasks  for  in-unit  testing.  Eighteen  (IS)  tasks  were  tested  at 
Fort  Campbell  after  a  19-week  interval  and  22  tasks  were  tested  at  Fort  Lewis 
after  a  21 -week  interval.  Since  many  of  the  tasks  had  retention  intervals  that 
exceeded  these  periods,  the  STAR  retention  performance  estimates  would  have 
been  low  and  were  adjusted  accordingly.  An.  additional  adjustment  was  made  in 
the  field  to  account  for  training  that  had  been  conducted  in  the  units.  In  all 
cases,  the  estimates  were  raised  prior  to  testing.  At  Fort  Campbell,  the  soldiers 
were  merely  tested  on  the  tasks.  The  Fort  Lewis  soldiers,  on  the  other  hand, 
were  tested  and  then  trained  to  one  perfect  GO.  They  were  then  given  two 
additional  trials  to  determine  if  the  first  GO  trial  was  due  to  chance. 

Findings: 

© 

1.  The  average  STAR  end-of-course  estimates  were  within  two 
percentage  points  of  the  average  observed  end-of-course  scores 
for  tasks  trained  to  criterion. 

© 

STAR^  was  calibrated  to  provide  retention  estimates  at  two 
levels  of  confidence.  At  the  99%  level  of  confidence,  it  was 
exDected  that  99%  of  the  observed  task  performance  data 
would  exceed  the  STAR2  estimates.  Thirty-eight  (3S)  out  of 
40,  or  95%,  of  the  tasks  met  this  expectation.  At  the  $0% 
confidence  level,  it  was  expected  that  S0%  of  the  observed  task 
performance  data  would  exceed  the  STAR"®  estimates.  Thirty- 
five  (3  5)  out  of  40,  or  37.5%,  of  the  tasks  met  this  expectation. 


3. 


For  these  first  trial  data,  STAR®  predicts  that  training  to 
criterion  one  GO  should  be  sufficient  exposure  to  sustain  skill 
level.  Ninety-one  (91)  percent  of  the  Fort  Lewis  soldiers  scored 
two  consecutive  perfect  trials  after  meeting  criterion  one  GO. 

These  soldiers  took  72%  fewer  sustainment  training  trials  to 
reach  criterion  than  they  had  required  to  learn  the  same  tasks 
in  the  training  base. 

Use  of  Findings: 

Criterion:  Training  to  the  STAR®  accuracy  criterion  yields  predetermined  pre¬ 
dictable  performance,  making  it  possible  to  trade  off  performance  versus 
training  cost. 

End-of-Course  Estimates:  The  ability  to  predict  EOC  performance  on  the  basis 
of  a  small  sample  of  first  trial  data  makes  it  possible  to  audit,  refine  and  de¬ 
velop  training  to  meet  predetermined  performance  objectives  in  the  training 
base. 

'S' 

Sustainment  Cycles:  Using  the  STAR'"'  calibration  feature,  it  is  possible  to 
generate  a  variety  of  skill  sustainment  strategies  and  plan  individual  skill 
sustainment  training  in  active  and  reserve  units. 

Refresher  Training  Model:  Training  to  criterion  one  GO  after  the  prescribed 
sustainment  interval  provides  sufficient  exposure  to  retrain  soldiers  in  units. 
This  retraining  represents  a  significant  saving  over  initial  training. 

Initial  Individual  Skill  Training  In-Units:  All  of  the  orocedures  used  in  this  ef¬ 
fort  can  be  transported  to  active  and  reserve  units  for  initial  individual  skill 
training. 
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EXECUTIVE  SUMMARY 


OBJECTIVES 

The  goal  of  this  mutual  effort  between  the  Army  Training  Board  (ATB)  and 
VIcFann,  Gray  3c  Associates,  Inc.,  (MGA)  was  to  perform  an  operational  test  of 
the  MGA  Skill  Training  and  Retention  (STAR  )  software  to  determine  if  it  can 
be  developed  into  a  useable  and  effective  tool  that  can  assist  field  commanders 
in  attaining  and  sustaining  high  levels  of  soldier  individual  skill  performance. 
This  test  had  three  objectives:  1)  To  test  the  ability  of  STAR®  software  to 
estimate  how  proficient  soldiers  would  be  after  training  in  the  training  base;  2) 
to  test  the  ability  of  STAR®  to  predict  the  decay  in  individual  skill  performance 
over  time  in  active  units;  and,  3)  to  determine  what  is  required  to  retrain  the 
individual  to  the  desired  proficiency  in  active  units. 

PROCEDURES 

Soldiers  from  platoons  of  two  new  manning  training  companies  participated  in 
this  effort.  Participants  received  standard  POI  task  information  and  demon¬ 
stration  and  were  then  tested  on  the  task.  First  trial  data  were  recorded  by  the 
task  force  and  transmitted  to  MGA  for  analysis. 

The  ATB  and  the  Infantry  School  (USAIS)  jointly  selected  114  CMF  11  (113,  C, 
and  H)  and  soldier's  common  tasks  for  this  test.  MGA  staff  produced  perfor¬ 
mance  measures.  Skill  Sustainment  Exercises  (SSEs),  that  were  jsed  for  testing 
and  training  the  tasks.  MGA  also  produced  a  General  Administrator's  Guide 
detailing  the  use  of  the  SSEs  and  specific  guides  for  each  SSE.  The  training  and 
data  collection  phases  of  this  operational  test  were  jointly  conducted  by  a  task 
force  nade  up  of  four  officers  from  ATB,  four  officers  of  USAIS,  and  two  N'G’Ds 
from  SQT  Management  Directorate.  Data  reduction  and  analysis  was  performed 
by  MGA  staff  nembers. 


In-Course  Phase 


Soldier  participants  received  standard  POI  introduction  information  and  task 
demonstrations.  They  were  then  tested  on  the  task  using  the  SSE.  A  soldier 
received  a  GO  on  a  task  step  if  it  was  correctly  performed.  When  the  soldier 
made  an  error,  he  was  given  a  NO  GO  on  the  step,  the  step  was  corrected,  and 
the  soldier  told  to  proceed.  No  task  instruction  was  given  beyond  correction  of 
the  error  step.  First  trial  data  were  recorded  by  the  task  force  and  transmitted 
by  telecommunication  to  \1GA  for  analysis. 

3ased  on  the  first  trial  GO/NO  GO  data,  for  each  task,  STAR®  provided  the  fol¬ 
lowing  estimates  which  were  telecommunicated  to  the  ATB  task  force: 

•  SSE  training  criterion:  The  number  of  successively  correct  (GO) 
trials  the  soldiers  were  required  to  perform  in  training  so  that 
S0%  had  stopped  making  errors.  (USAIS  selected  this  S0%  level 
criterion.) 

•  STAR®  estimate  of  the  minimum  End-of-Course  performance. 

•  STAR®  estimate  of  the  retention  interval  in  weeks  for  skill  decay 
to  60%  correct  performance. 

The  soldiers  were  then  trained  to  criterion  on  the  SSEs  and  returned  to  the  PCI. 
SSE  training  and  testing  was  scheduled  so  that  it  did  not  interfere  with  the 
administration  of  the  standard  POI. 

Prior  to  the  regularly  scheduled  end-of-course  (EOC)  refresher  training,  the  task 
force  members  tested  the  soldiers  to  determine  their  EOC  proficiency  on  each  of 
the  tasks. 


\T3  select  »d  tasks  for  :n-unit  testing.  Twenty-: 
Fort  Campoeii  if  ter  a  19-week  interval.  Four  of  t 
day  before  the  ta-:k  force  arrived  for  testing:  as 
eliminated  from  the  analysis.  As  excected.  the  sold 


:wo  (12)  tasks  were 
hese  tasks  were  tr 
i  result,  these  ta 
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r,o  errors 


on  these  tasks.  Twenty-two  (22)  tasks  were  tested  a"  Fort  Lewis  after  a  7 ! -week 
interval.  Since  many  of  the  tasks  had  long  retention  ntervals  (e.g.,  one  to  three 
years),  it  was  logistically  impossible  to  wait  untii  the  STAR®  interval  had 
expired.  Accordingly,  minimum  performance  estimates  were  made  using 
STAR®  software  adjusted  to  the  field  test  intervals.  Minimum  performance 
estimates  were  also  adjusted  (upward)  to  account  for  any  training  that  had  been 
conducted  in  the  units. 

At  Fort  Campbell,  the  soldiers  were  tested  by  administering  the  SSEs  one  time. 
At  Fort  Lewis,  the  soldiers  were  tested  and  the  data  recorded  for  their  first 
attempt  at  each  task.  Then  the  soldiers  were  trained  until  they  performed  each 
task  step  correctly.  They  were  then  given  two  additional  trials  to  determine  how 
many  soldiers  were  performing  the  task  correctly  after  attaining  one  correct 
trial.  All  testing  and  data  collection  were  performed  by  the  ATft  task  force. 

FINDINGS 

STAR®  End-of-Course  Estimates 

The  average  STAR®  end-of-course  estimates  were  within  two  percentage  points 
of  the  average  observed  end-of-course  scores  for  tasks  trained  to  criterion. 

STAR®  Retention  Estimates 

The  STAR®  minimum  performance  estimates  for  the  in-unit  test  were  based 
upon  two  levels  of  calibration  confidence.  The  confidence  limits  were  99%  and 
30%  respectively.  For  the  99%  confidence  limit,  it  was  expected  that  '^9%  of 
the  observed  task  performance  data  would  exceed  the  ST  AR®  estimates.  Thirty - 
eig'-'t  (33)  out  of  ^0,  or  95%,  of  the  tasks  met  this  expectation.  For  the  39 % 
confidence  limit,  it  was  expected  that  30%  of  the  observed  task  performance 
data  would  exceed  the  STAR®  estimates.  Thirty-five  (35)  out  of  -3,  or  3%  5%.  of 
the  tasks  net  this  expectation. 


Sustainment  Training  Model 


MGA's  previous  experience  in  process  manufacturing  operator  training  had  shown 
that  refresher  training  to  a  criterion  of  one  GO  was  sufficient  exposure  for  skill 
sustainment.  This  sustainment  training  method  was  tested  at  Fort  Lewis. 

The  Fort  Lewis  soldiers  were  tested  and  then  trained  to  criterion  one  perfect 
GO,  they  were  then  given  two  additional  trials  and  their  response  accuracy 
recorded.  Ninety-one  (91)  percent  of  the  soldiers  scored  two  consecutive  perfect 
trials  after  meeting  criterion  one  GO,  indicating  they  had  been  effectively 
retrained. 

The  soldiers  required  72 %  fewer  sustainment  training  trials  to  reach  criterion 
than  the  same  soldiers  had  required  learning  the  same  tasks  in  the  training  base. 

USE  OF  THE  FINDINGS 

STAR®  software  can  be  used  to  provide  the  following  products  which  have 
application  for  unit  and  school  training.  These  products  are  illustrated  in  the 
figure  on  page  6.  The  discussion  below  is  keyed  to  the  figure  by  Roman 
numerals. 

Criterion  (I) 


Training  to  the  STAR®  accuracy  criterion  yields  predetermined  predictable 
performance,  making  it  possible  to  trade  off  performance  versus  training  cost. 

End-of-Course  Estimates  (ID  and  Training  Recommendations  (III) 

The  ability  to  predict  EOC  performance  on  the  basis  of  a  small  sample  of  first 
trial  data  makes  it  possible  to  audit,  refine  and  develop  training  to  meet 
predetermined  performance  objectives  in  the  training  base. 


Using  the  STAR®  calibration  feature,  it  is  possible  to  generate  a  variety  of  skill 
sustainment  strategies  and  plan  individual  skill  sustainment  training  in  active  and 
reserve  units.  For  example,  these  planning  strategies  might  emphasize  task 
criticality,  integration  of  individual  skill  sustainment  in  collective  training 
exercises  or  cross  training. 

Refresher  Training  (V) 

Training  to  criterion  one  GO  provides  sufficient  exposure  to  retrain  soldiers  in 
units.  This  retraining  represents  a  significant  saving  over  initial  training. 

In-Unit  Initial  Individual  Skill  Training  (V) 


All  of  the  procedures  used  in  this  effort  can  be  transported  to  active  and  reserve 
units  for  initial  individual  skill  training.  This  training  will  yield  predictable 
results  on  tasks  that  are  not  trained  in  the  training  base  as  well  as  new  weapon 
training  tasks  and  career  advancement  training. 


SE  PRODUCTS  DEVELOPED  JN  THIS  OPERATIONAL  TEST 


I.  INTRODUCTION 


BACKGROUND 

The  Army  is  required  to  maintain  a  stabilized,  ready  force.  Integral  components 
of  this  requirement  are:  modernization  of  the  force;  mobilization  preparation  of 
the  force;  and  maintenance  of  known  force  capability  for  immediate  response  to 
threat.  To  ensure  the  success  of  this  mission,  the  Army  must  train  and  sustain 
individual  and  collective  soldier  skills  at  specified  levels.  The  Army  Training 
Board  (ATB)  has  initiated  a  large  scale  program  with  the  goal  of  determining 
training  frequencies  and  evaluation  cycles  for  individual  and  collective  skills  in 
both  active  and  reserve  components.  This  report  addresses  one  component  of 
this  program  -  the  sustainment  of  individual  skills  at  required  levels  of 
proficiency.  Specifically,  this  report  summarizes  the  results  of  an  operational 
test  to  verify  the  ability  of  existing  technology  to  estimate  training  and  skill 
retention  outcomes  in  Army  schools  and  units. 

The  test  came  about  for  two  reasons.  First,  the  Army  has  a  requirement  to  train 
and  maintain  specified  performance  levels  for  individual  soldier  tasks,  but  the 
Army  has  no  validated  tool  to  estimate  minimum  skill  training  and  retention 
proficiency. 

Second,  McFann,  Gray  A:  Associates,  Inc.,  (MGA)  had  developed  and  validated 
analytic  computer  software  that  permitted  such  estimation  for  individual  tasks 
in  non-military  educational  and  process  manufacturing  training  settings.  How¬ 
ever,  the  MGA  Skill  Training  and  Retention0  (STAR)  software  had  not  been 
validated  for  Army  tasks. 

As  a  result  of  these  two  considerations,  it  was  decided  to  conduct  a  oilot 
operational  test  of  STAR  in  a  military  training  base.  This  study  was  conducted 
as  an  engineering  operational  test  and  verification  of  the  usefulness  of  STAR.  It 
was  not  a  basic  research  study. 


II.  APPROACH 


PILOT  OPERATIONAL  TEST 

V1GA  and  ATE  conducted  a  pilot  test  in  which  STAR  was  used  to  estimate  end- 
of-course  (EOC)  scores  for  three  Artillery  OSUT  tasks  at  Fort  Sill.  The  tasks  by 
MOS  were: 

•  13-B:  Emplace  and  recover  the  collimator  (collimator) 

Prepare  semifixed  ammunition  (ammunition) 

•  13-E:  Prepare  a  surveyed  firing  chart  (firing  chart) 

The  STAR  software  program  analyzed  the  soldiers'  first  trial  performance  data 
and  provided  an  estimate  of  the  performance  score  that  would  result  at  EOC. 
The  estimated  EOC  scores  represented  the  lowest  performance  which  we 
expected  on  each  of  the  tasks.  Therefore,  we  expected  the  actual  field  data 
would  equal  or  exceed  our  estimated  score  if  the  software  was  valid  for  Army 
tasks.  The  estimates  of  EOC  scores  were  valid  as  shown  in  Table  1. 

TABLE  l:  PILOT  OPERATIONAL  TEST  RESULTS 


TASK 

TRAINING  READINESS 

MGA  SOFTWARE 
ESTIMATE 

ACTUAL 
FIELD  DATA  I 
.  1 

Emplace  and  Recover 
the  Collimator 

9U% 

1 

1 

95%  j 

Prepare  Semi-Fixed 
Ammunition 

90 % 

92c6  | 

Prepare  a  Surveyed 

Firing  Chart 

S0°o 

?1  6 

3ased  upon  the  demonstrated  utility  of  the  MGA  software  for  estimating  EOC 
performance  for  these  artillery  tasks  from  initial  performance,  the  decision  was 
made  to  undertake  a  large  scale  verification  of  the  \1GA  software  for  both  EOC 
and  retention  intervals.  If  the  data  held  up  and  indications  were  that  it  had 
practical  utility,  then  the  Army  would  have  made  an  important  step  forward  in 
providing  a  tool  to  units  in  the  field  to  plan  individual  training  based  upon  skill 
retention  and  skill  sustainment. 

SUPPLEMENTARY  VALIDATION  AND  DATA  BASE  DEVELOPMENT 

Overview 

The  supplementary  validation  was  conducted  in  two  phases.  The  training  base 
phase  was  conducted  at  the  U.S.  Army  Infantry  School  (USAIS),  Fort  Benning, 
and  retested  the  STAR  capability  to  estimate  minimum  performance  scores.  The 
in-unit  phase  was  conducted  at  Forts  Campbell  and  Lewis  and  tested  the 
capability  of  STAR  to  estimate  minimum  skill  retention  scores. 

The  ATB  and  the  Infantry  School  initially  selected  CMF  11  and  Soldier's  Common 
Tasks  that  are  critical  to  a  broad  scope  of  the  Army's  mission  for  this  test.  The 
testing  phase  of  the  study  was  jointly  conducted  by  a  task  force  of  four  officers 
from  ATB  (including  the  COTR),  four  officers  of  USAIS  and  two  NCOs  from  SQT 
Management  Directorate,  and  MGA  staff  researchers. 

The  following  operational  decisions  were  reached: 

•  First,  ATB  and  MGA  would  conduct  this  effort  as  a  joint  activity. 

•  Second,  the  tasks  selected  for  study  would  be  tasks  that  are  critical  to 
a  broad  scope  of  the  Army's  mission.  The  USAIS  and  ATB  selected  114 
tasks.  These  tasks  were  drawn  from  the  Common  Tasks,  MOS  I  IB,  1 1C. 
1 1  H  and  AS  E  9  and  C2. 

•  Third,  MGA  would  train  the  AT3  task  force  in  the  appropriate 
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technology  for  obtaining  initial  soldier  performance,  FOC  performance, 
and  in-unit  performance. 


•  Fourth,  the  AT3  task  force  would  conduct  both  the  criterion  training 
and  verification  testing.  Task  introduction  and  task  demonstration  was 
conducted  by  USA1S  committee  instructors. 

•  Fifth,  MGA  would  provide  estimates  of  FOC  performance  and  perfor¬ 
mance  score  estimates  for  various  retention  intervals.  These  estimates 
would  be  based  upon  soldiers'  initial  performance  on  the  tasks  and 
derived  from  the  MGA  computer  software. 

•  Sixth,  a  user's  guide  or  handbook  would  be  prepared  which  included,  for 
each  CMF  11  task,  tested  information  on  the  frequency  and  amount  of 
refresher  training  required  to  maintain  performance. 

•  Seventh,  in  addition  to  validating  the  MGA  computer  program,  MGA 
and  ATS  would  select  those  tasks  and  variables  judged  most  related  to 
skill  retention  and  decay  and  generate  a  composite  score  using  those 
tasks.  Thus,  tasks  would  be  categorized  according  to  the  number  of 
features  they  have  that  are  thought  to  contribute  to  skill  decay.  The 
idea  was  to  determine  if  it  was  possible  with  these  categories  to 
provide  a  readily  useable  prediction  of  those  tasks  most  susceptible  to 
skill  decay  and  therefore  requiring  more  frequent  sustainment  or 
refresher  training. 

The  focus  of  this  study  has  been  to  test  existing  technology  in  a  field  setting, 
with  the  emphasis  on  spelling  out  what  actions  are  required  to  permit  application 
of  the  technology  in  the  field.  The  aim  is  utilization  of  existing  technology 
through  adoption  or  adaption  to  Army  requirements,  not  generation  of  new 
technology.  The  remainder  of  this  report  details  the  method  used,  the  results  of 
the  study,  and  the  imolications  for  applying  the  tested  technology  within  the 
Army. 


pis2ure  2  is  a  graphic  overview  of  the  operational  test  sequence 


1'IGllur.  2:  PROGRAM  OPERATIONAL  TEST  SEOI  lENfM- 


The  major  milestones  are  numbered  boxes  1.0  through  4.0.  They  are: 

1.0  Information  Delivery.  All  of  the  soldier  participants  received  the 
standard  POI  introduction  and  demonstration  prior  to  their  first 
practice  opportunity  on  each  task. 

2.0  First  Trial  Test.  Soldier  participants  were  then  separated  from  the 
non-participants  and  given  a  test  on  the  task.  These  data  were 
analyzed  using  the  STAR  software  and  three  estimates  resulted: 

2.1  Criterion.  The  number  of  successive  correct  training 
trials  necessary  to  ensure  that  S0%  of  the  soldiers  had 
stopped  making  errors  in  training. 

2.2  End-of-Course  Prediction.  Based  on  first  trial  data, 
STAR  estimated  the  minimum  end-of-course  performance 
for  each  task. 

2.3  In-Unit  Prediction.  Based  on  first  trial  data,  STAR 
estimated  the  interval  over  which  performance  would 
decay  to  60%  in  the  absence  of  training  or  practice. 

3.0  Training.  Training  was  conducted  according  to  four  procedures 
that  will  be  discussed  in  a  following  section. 

4.0  End-of-Course  Testing.  Prior  to  the  normally  scheduled  end-of- 
course  POI  reinforcement  training,  soldiers  were  given  a  test  on 
each  task  for  which  they  had  received  training.  The  EOC  results 
were  compared  to  the  STAR  predictions  to  test  the  validity  of 
these  estimates. 

U0  In-Unit  Training  and  Testing.  ST \R  sofware  allows  the  user  to 

calibrate  retention  predictions  for  any  arbitrary  retention  mtervnl. 
5T\R  will  toen  generate  estimates  of  minimum  performance  at 


selected  intervals.  Soldiers  were  tested  after  19  weeks  at  Fort 
Campbell  and  after  21  weeks  at  Fort  Lewis.  The  ST AR  in-uni t 
estimates  were  adjusted  both  for  these  intervals  and  for  unit 
training  that  had  taken  place  since  graduation  from  the  training 
base  (see  Figure  2,  2.3  and  2.3.1).  In  these  instances,  the  estimates 
were  revised  upward,  and  the  revisions  were  made  prior  to  testing. 
The  in-unit  results  were  compared  to  the  STAR  predictions  to  test 
the  validity  of  these  estimates. 

Skill  Sustainment  Exercises 

STAR  requires  a  stev -by-step  performance  measure  for  each  task.  To  obtain 
these  data  for  each  task,  \1GA  staff  members  developed  data  gathering 
instruments  called  Skill  Sustainment  Exercises  (SSEs).  SSEs  provided  a  consis¬ 
tent  means  of  measuring  task  performance  before  and  after  training.  SSEs  were 
used  to  collect  the  first  trial,  EOC,  and  in-unit  performance  data. 

The  SSEs  were  developed  by  the  MCA  staff  who  performed  a  front-end  analsyis 
of  each  of  the  1 44  tasks  that  were  included  in  the  study.  Inputs  for  this  analysis 
were:  soldier's  manuals,  technical  manuals,  field  manuals,  existing  task  analyses, 
job  aids,  training  aids,  POIs  and  POIQTs  for  each  task,  and  other  relevant 
government  publications.  The  front-end  analysis  consisted  of  an  allocation  of 
functions  between  the  soldier  and  the  equipment,  a  task  analysis  that  detailed 
the  steps  required  to  perform  the  task,  and  a  behavioral  task  analysis  that 
clustered  and  sequenced  task  steps  according  to  the  relevance  of  available 
learning  cues.  For  some  short  tasks,  this  task  analysis  provided  the  final  SSE. 
For  longer  tasks,  redundant  task  steps  were  eliminated  to  prepare  the  SSE.  The 
criteria  for  all  SSEs  were: 

•  Each  SSE  step  required  a  single  action. 

•  The  action  or  result  of  the  action  nust  he  observable. 

•  The  action  nust  he  scoreable  as  a  "GO"  or  a  "NO  GO." 


•  The  action  must  follow  a  logical  sequential  progression  toward  the 
achievement  of  the  task  goal. 

•  The  SSE  learning  sequence  must  be  consistent  with  current  POI  training 
sequence. 

In  addition  to  the  SSEs,  MG  A  developed  a  general  administrator's  guide  to  train 
the  :usk  force  in  the  SSE  data  collection  and  training  procedures.  For  each  SSE, 
M  SA  developed  an  SSE  Requirements  Sheet  that  detailed  the  personnel,  equip- 
’(m'’ :,  and  procedures  required  to  conduct  the  training  and  data  collection.  The 
SSE  task  and  training  development  steps  are  illustrated  in  Figure  3.  Examples  of 
all  SSE  materials  are  contained  in  Volume  III. 


FRONT  END  ANALYSIS 


SSE  TASK  DEVELOPMENT 


FUNCTIONAL 

ALLOCATION 


SSE  TRAINING  DEVELOPMENT 


POI  ANALYSIS 


IN-COURSE  PHASE 


Training  of  Data  Collectors  and  Trainers 

The  data  collectors  and  trainers  were  trained  at  Fort  Benning  during  the  two 
weeks  prior  to  the  start  of  data  collection.  Two  MG  A  staff  members  traveled  to 
Fort  Benning  to  provide  quality  assurance  and  coordination  during  this  start-up 
period.  The  data  collectors  and  trainers  were  trained  to  administer  and  score 
the  SSEs  to  ensure  accurate  data  for  input  to  STAR. 

Prior  to  training,  the  Infantry  School,  in  coordination  with  the  ATB  task  force, 
was  required  to  specify  the  SSE  training  criterion.  This  criterion  is  an  interim 
standard  and  specifies  the  level  of  accuracy  the  soldiers  have  to  achieve  before 
returning  to  the  POI  for  Soldier  Manual  Standard  Training.  The  School  selected 
an  accuracy  of  30%  GO.  This  meant  that  the  soldiers  would  be  trained  using  the 
SSE  until  30%  of  them  had  stopped  making  errors.  (At  least  eight  of  ten  soldiers 
would  be  able  to  perform  the  task  100%  correctly  on  the  next  trial.) 

The  soldiers  received  current  POI  orientation  and  a  demonstration  of  each  task 
from  the  Infantry  Training  Group  Cadre  or  the  ATB  task  force.  At  the  first 
opportunity  to  practice  each  task,  the  soldiers  were  removed  from  POI  training 
and  required  to  perform  the  task.  Each  soldier  was  scored  on  the  SSE  for  the 
first  trial.  When  a  soldier  received  a  NO  GO  on  an  SSE  step,  he  was  immediately 
stopped,  informed  of  the  correct  step,  and  required  to  perform  the  step  correctly 
before  continuing.  These  data  were  electronically  communicated  to  MGA  for 
ST  \R  software  analysis. 

MGA  analyzed  the  first  trial  data  and  provided  the  following  information: 

•  The  number  of  successively  correct  trials  the  soldiers  would  save  to 
perform  for  the  software  to  indicate  that  the  30%  triterion  was  net. 

•  \n  estimate  of  tire  lowest  expected  performance  score  at  the  end  of 
course. 


•  The  time  interval  (in  weeks  after  the  course)  in  which  soldier 
proficiency  would  decay  to  60%. 

The  study  design  and  the  STAR  software  requirements  called  for  soldiers  to  be 
trained  to  SSE  criterion  following  the  initial  trial  and  then  returned  to  the  POI 
for  training  to  soldier  manual  standard.  However,  since  this  study  was 
superimposed  on  the  POI  training,  the  training  fell  into  four  categories.  The 
soldiers  were: 

•  Trained  to  STAR  SSE  criterion  and  to  soldier  manual  standard 
(Training  Method  1) 

•  Trained  to  STAR  SSE  criterion  only  (Training  Method  2) 

•  Trained  to  soldier  manual  standard  only  (Training  Method  3) 

•  Initial  trial  data  collected  and  retested  without  intervening  training 
(Training  Method  4). 

Only  Training  Method  1  met  all  the  STAR  training  requirements  for  optimal 
estimation  accuracy.  The  other  three  methods  each  lacked  some  of  the 
requirements,  thereby  reducing  expected  estimation  accuracy. 

SSEs  were  used  for  pre-  and  post-testing  under  all  four  training  methods. 
Training  was  performed  by  the  task  force  to  minimize  turbulence  in  current  POI 
training.  The  training  and  testing  cycle  is  illustrated  in  Figure  4. 

Subjects 

Two  platoons  of  trainees  were  selected  from  two  training  companies.  One 
company  ■'■as  beginning  training  and  the  other  was  in  the  seventh  week  of  the 
POI.  This  overlap  allowed  the  task  force  to  collect  all  data  in  approximately 
eight  weeks.  Table  2  shows  the  number  of  trainees  who  participated  in  all  phases 
jf  the  studv  Un-Course  and  In-Unit)  bv  MOS  and  advanced  skill  tasks. 
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TABLE  2:  SOLDIER  PARTICIPANTS 


Number 

of 

Soldiers 

Training  Tasks 

16  1  IB  AND  COMMON  TASKS 

20  VI 60 ,  NIGHT  SIGHTS  AND 

STT  TASKS 

13  11C  TASKS 

S  11H  (TOW)  TASKS 

_10 _  1  IB  (DRAGON)  TASKS  (ASI  E9) 

67  TOTAL 


Age/ 

Years 

Education/ 

Years 

GT 

Score 

Mean 

20.46 

11.75 

107.15 

S.D. 

2.93 

0.93 

12.51 

Range 

17  to  2S 

9  to  14 

74  to  143 

STAR  Software  Data  Requirements 


The  MGA  software  requires  a  minimum  of  200  GO/NO  GO  data  points  per  task  to 
yield  stable  estimates.  In  general,  the  distribution  of  subjects  fulfilled  this 
requirement.  Tasks  that  did  not  meet  this  requirement  are  noted  in  the  data 
appendix,  Volume  II. 

The  STAR  software  flags  tasks  in  which  the  estimated  EOC  variability  and 
observed  EOC  variability  differ,  i.e.,  when  the  observed  error  is  greater  than 
expected.  When  a  task  is  flagged  in  this  way  it  is  necessary  to  re-analyze  the 
total  training  and  data  collection  to  enable  ST  AR  to  be  used  to  estimate 
requirements  for  that  task. 

End-of-Course  Data 


The  ATE  task  force  used  the  SSEs  to  collect  end-of-course  data  prior  to  PDI 
reinforcement  training.  These  data  were  electronically  transmitted  to  MGA  for 
analysis.  Daper  topies  of  ail  of  the  raw  data  collected  during  the  study  were 
sent  to  MGA  to  ensure  the  accuracy  of  the  electronically  transmitted  data. 


IN-UNIT  PHASE 


Overview 


The  soldiers  who  were  tested  at  Fort  Benning  were  in  New  Manning  System  units 
which  went  to  Fort  Campbell,  Kentucky  and  Fort  Lewis,  Washington.  They  were 
re-tested  on  selected  tasks  to  validate  STAR  estimates  of  task  performance 
after  specific  retention  intervals.  Twenty-two  tasks  were  tested  at  Fort 
Campbell  19  weeks  after  the  completion  of  Fort  Benning  training.  Twenty-two 
additional  tasks  were  tested  at  Fort  Lewis  21  weeks  after  the  completion  of  Fort 
Benning  training.  These  intervals  were  chosen  as  the  best  trade-off  between 
theoretical  retention  decay  times  and  unit  schedules  to  minimize  unit  tur¬ 
bulence. 

The  STAR  retention  estimates  were  based  on  the  time  interval  between  training 
and  retention  testing.  Therefore,  they  were  revised  to  take  into  account  any 
refresher  training  that  occurred  between  the  end  of  the  course  and  the  retention 
test  date. 

Fort  Campbell  Data  Collection  Procedures 

The  ATB  task  force  collected  data  using  the  SSEs  that  were  administered  during 
the  training  phase.  Each  soldier  was  given  the  SSE  one  time  and  GO/NO  GO  data 
were  recorded  for  each  performance  measure  of  the  SSE.  The  22  tasks  w'ere 
distributed  across  M60,  TOW,  DRAGON,  and  MORTAR  weapon  systems.  In-unit 
refresher  training  had  been  conducted  for  many  of  the  tasks  that  were  tested. 
The  dates  and  extent  of  this  training  were  collected  from  the  company  training 
NCO  and  company  commander,  ^s  it  turned  out,  all  but  the  TOW  tasks  had 
received  refresher  training  during  the  19  week  interval.  The  Fort  Campbell 
tasks  are  listed  in  Table  '•*.  The  complete  data  displays  are  presented  in  Volume 
II,  \ppendix  3. 


Fort  Lewis  Data  Collection  Procedures 


With  one  exception,  the  Fort  Lewis  data  collection  procedures  were  identical  to 
those  employed  at  Fort  Campbell.  All  22  tasks  were  tested  for  retention 
proficiency,  tasks  were  then  trained  to  criterion  3  correct  trials.  The  results  of 
this  criterion  training  will  be  discussed  in  a  later  section  of  this  report.  All  22 
tasks  tested  at  Fort  Lewis  were  selected  from  the  Soldier's  Common  Tasks.  The 
Fort  Lewis  tasks  are  listed  in  Table  5.  The  complete  data  displays  are  presented 
in  Volume  II,  Appendix  3. 


TABLE  4:  FORT  CAMBPELL  TASKS 


071-321-1601  Prepare  an  M16  Plotting  Board  for  Operation  (Pivot  Point) 


071-321-1602 


Process  FO  Corrections  Using  an  Ml 6  Plotting  Board  (Pivot 
Point) 


07 1-321-3901  Place  a  Ground  Mounted  Si  mm  Mortar  Into  Action 


071-321-3904 


Lay  Mortar  for  Deflection  and  Elevation  (D&E)  (Ground/Carrier 
Mounted) 


071-321-3909  Refer  Sight  and  Realign  Aiming  Posts 


071-321-3910 


Reciprocally  Lay  Mortar  Using  M2  Aiming  Circle  and  Place  Out 
Aiming  Posts 


071-316-2521  Prepare  an  Antiarmor  Range  Card 

071-317-0000  Prepare  an  Antiarmor  Range  Card  (DRAGON) 

071-317-3301  Conduct  a  Pre-operational  Inspection  of  the  DRAGON  Tracker 
and  Round 

071-317-3302  Prepare  the  DRAGON  for  Firing 
071-317-3303  Determine  if  a  Target  is  Engageable  (DRAGON) 

071-317-3304  Demonstrate  Correct  DRAGON  Firing  Positions 
071-317-3306  Perform  Immediate  Action  Procedure  for  a  DRAGON  Misfire 
071-317-3308  Perform  Emergency  Destruction  Procedures  on  a  DRAGON 
071-312-3001  Load,  Reduce  a  Stoppage,  and  Clear  the  M60  Machinegun 
071-316-2500  Assemble  the  TOW  Launcher 

071-316-2501  Perform  Operator  Maintenance  on  a  TOW  Launcher 

071-316-2502  Conduct  System  Self  Test  and  Pre-operational  Inspection  of 
TOW  Launcher 

071-3  16-2503  Load,  \rm,  and  Unload  an  Encased  TOW  Missile 
071-316-250^  Perform  Immediate  -\ction  for  a  TOW  Misfire 
071-3  i  6-2505  Determine  if  a  Tars**:  can  he  Engaged  b>  TOW 


071-316-2516  Determine  TO\\  Firing  Limitot;  sns 


TABLE  5:  FORT  LEWIS  TASKS 


071-316-0814  Identify  Friendly  and  Threat  Armored  Vehicles 


071-318-2201 


071-318-2203 


Prepare  an  M72A2  LAW  for  Firing;  Restore  Y172A2  LAW  to 
Carrying  Configuration 

Apply  Immediate  Action  to  Correct  a  Malfunction  on  an  M72A2 
LAW 


071-329-1001  Identify  Terrain  Features 

071-329-1002  Determine  the  Grid  Coordinates  of  a  Position  Using  the 
Ylilitary  Grid  Reference  System 

071-329-1003  Determine  a  Magnetic  Azimuth  Using  a  Compass 


051-192-1002 


051-192-1 


Install  an  M16A1  Mine  Bounding  Antipersonnel  Mine  without 
Tripwires 

Install  M21  Metallic  Antitank  (AT)  Mine 


051-192-1022  Locate  Mines  by  Probing 

051-192-1502  Install  and  Fire/Recover  an  M1SA1  Claymore  Mine 
081-831-1013  Give  First  Aid  to  a  Blood-Agent  Casualty 
031-831-1011  Give  First  Aid  to  a  Nerve-Agent  Casualty 
05 1-192-1007A  Decontaminate  Your  Skin 


031-503-1 


Give  the  Alarm  for  a  Chemical  or  Biological  (CB)  Hazard 


031-503-1 


Put  on  and  Wear  an  M  17-Series  Protective  Mask 


071-311-2 


071-311-2 


Load,  Reduce  a  Stoppage,  and  Clear  an  M16A1  Rifle 

Perform  Operator  Maintenance  on  an  M16A1  Rifle,  Magazine, 
and  Ammunition 


081-831-1001  Perform  Mouth-to-Mouth  Resuscitation 


081-831-1005  Prevent  Shock 


081-331-1006  Splint  a  Suspected  Broken  Arm  or  Leg 


081-831-1016 


Put  on  a  Field  or  Pressure  Dressing 


071-331-0803  Coilect/Report  Information  (SALUTE) 


in.  STAR  RESULTS 


The  results  will  be  described  separately  for  the  In-Course  and  In-Unit  study 
phases.  In  both  phases  the  results  center  around  the  ability  of  STAR  to  make 
useful  EOC  and  retention  performance  estimates. 


IN-COURSE 


EOC  performance  scores  were  estimated  based  on  initial  task  performance.  The 
tasks  were  then  trained  under  one  of  the  four  training  methods  described  in 
Section  II.  The  major  results  of  the  In-Course  Phase  are  presented  in  Table  6. 
The  findings  are: 

•  Average  STAR  EOC  estimates  were  within  two  (2)  percentage 
points  of  the  average  EOC  observed  scores  for  the  criterion- 
trained  tasks  (Training  Method  1  -  average  estimate  =  95%, 
average  observed  =  93%;  Training  Method  2  -  average  estimate 
=  92%,  averaged  observed  =  91%). 

•  Average  EOC  scores  were  greater  for  tasks  which  were  trained 
to  criterion,  93%  and  91%  (Training  Methods  1  and  2),  than  for 
those  not  trained  to  criterion,  85%  and  80%  (Training  Methods  3 
and  4). 

•  A  greater  percentage  of  tasks  that  were  trained  to  criterion 
exceed  80%  and  90%  EOC  scores  than  tasks  not  trained  to 
criterion. 


The  task-by-task  data  are  contained  in  Volume  II,  Appendix  3. 


TABLE  6:  SUMMARY  OF  EOC  RESULTS 


T  raining 
Method 

Number 

Tasks 

Average 

STAR 

Estimate 

Average 

Observed 

Score 

Number  of 
Observed 
Task  Scores 
Z>S0% 

Number  of 
Observed 
Task  Scores 
=>90% 

1 

7 

95% 

93% 

7  ( 100%) 

7  (100%) 

-> 

L. 

29 

92% 

91% 

27  (93%) 

19  65%) 

3 

12 

93% 

S5% 

7  (58%) 

5  (42%) 

4 

65 

93% 

S6% 

49  (75%) 

29  (45%) 

Revision  of  the  STAR  Procedure 

The  Statement  of  Work  included  provisions  for  practical  revision  of  STAR  based 
upon  field  data  and  experience.  No  revisions  of  the  STAR  sofware  were 
necessary;  however,  the  ATB  task  force  found  that,  given  available  staff  and 
resources,  it  was  impractical  to  train  to  the  STAR  criterion  in  many  instances. 
MGA  and  the  COTR  agreed  on  a  field  expedient  revised  criterion  based  on  the 
first  trial  data.  The  rules  for  this  criterion  are  shown  in  Table  7. 

TABLE  7:  FIELD  EXPEDIENT  CRITERION  RULES 


First  Trial 

Field 

Error 

Criterion 

<-  5% 

3 

<J0%E>5% 

6 

C>io% 

STAR  Value 

Far  example,  a  task  whose  average  correct  performance  was  91%  would  be 
trained  to  six  successively  correct  trials. 


These  rules  we  re  developed  about  half-way  through  the  data  collection  and, 
therefore,  were  not  universally  applied.  Data  were  collected  on  32  tasks  in 
which  the  field  expedient  criterion  was  used.  The  average  criterion  used  was  3.5 
successively  correct  trials.  The  STAR  estimate  for  this  criterion  is  that  of 
the  soldiers  would  stop  making  errors  with  this  criterion.  The  STAR  criterion  for 
30%  error  free  performance  was  3.1  successively  correct  trials  for  the  same 
tasks.  The  impact  of  the  lower  criterion  was  not  evaluated  in  this  effort. 
However,  STAR  simulations  show  that  the  lower  criterion  is  a  source  of 
variability.  This  issue  should  be  addressed  in  future  applications  of  STAR 
technology. 

IN-UNIT 

Two  types  of  STAR  retention  estimates  were  made.  The  first  estimate  is  the 
interval  of  time  over  which  performance  will  decay  to  60%.  The  60%  strategy 
was  selected  by  ATS  for  this  study.  The  second  estimate  is  an  estimate  of 
retention  performance  for  various,  specified  intervals  of  time  following  training 
or  refresher  training.  The  first  estimate  (i.e.,  60%  estimate)  was  based  on 
initial  performance.  The  60%  retention  intervals  ranged  from  a  few  weeks  to 
more  than  three  years.  It  would  have  been  logistically  difficult  to  verify  STAR 
estimates  following  each  of  these  intervals.  Therefore,  the  second  estimates 
were  made  based  on  the  availability  of  the  test  soldiers  at  Fort  Campbell  and 
Fort  Lewis,  19  and  21  weeks  after  EOC,  respectively.  Minimum  performance 
estimates  were  generated  using  STAR  for  each  task  based  on  the  19  and  21  week 
retention  intervals  between  training  and  testing.  At  Forts  Campbell  and  Lewis, 
company  officers  and  NCOs  were  interviewed  to  determine  which  of  the  tasks 
had  received  unit  refresher  training.  This  information  was  analyzed  by  STAR  to 
further  revise  the  STAR  performance  scores.  In  all  cases,  the  estimates  were 
revised  upward  orior  to  retention  testing  at  Fort  CamDbell  and  Fort  Lewis. 
These  task-bv-task  estimates  are  contained  in  Volume  II,  Appendix  3. 

STAR  retention  estimates  generated  for  this  study  are  estimates  of  the 
minimum  performance  that  ■'.ill  occur  after  a  specified  retention  interval.  The 
''T  AR  software  produced  two  estimates  for  retention  performance  in  each  task: 


80%  and  99%  calibration  levels.  The  80%  calibration  provides  the  higher 
minimum  performance  estimates  of  the  two,  but  runs  a  greater  risk  (20%)  that 
the  actual  retention  score  may  fall  below  the  estimate.  The  99%  calibration 
provides  a  lower  minimum  performance  estimate,  and  consequently,  lower  risk 
(1%)  that  retention  performance  will  fall  below  the  estimate.  The  retention 
performance  estimates  were  made  for  the  specific  interval  which  had  passed  for 
each  task  since  it  had  been  trained  or  retrained. 

The  results  of  the  In-Unit  Phase  are  summarized  in  Table  8  and  Figure  o.  The 
major  findings  were: 

•  For  the  80%  STAR  calibration,  retention  performance  on  35  of  40,  or 
87.5%,  of  the  tasks  met  or  exceeded  the  STAR  minimum  performance 
estimates.  The  frequency  distribution  of  the  actual  scores  minus  the 
estimated  scores  is  shown  in  Figure  5. 

•  For  the  99%  STAR  calibration,  retention  performance  on  38  of  40,  or 
95%  of  the  tasks  met  or  exceeded  STAR  minimum  perforinan.ee 
estimates.  The  frequency  distribution  of  the  actual  scores  minus  the 
estimated  scores  is  shown  in  Figure  5. 

•  The  average  absolute  difference  between  estimated  and  actual  scores 
was  9  percentage  points  at  the  80%  calibration  and  21  percentage 
points  at  the  99%  calibration. 

The  complete  task-by-task  data  are  contained  in  Volume  II,  Appendix  3. 


As  noted  earlier,  soldiers  at  Fort  Lewis  were  not  only  tested,  they  were  also 
trained  to  criterion.  This  part  of  the  study  was  an  offshoot  of  previous  work  by 
MGA  staff.  MGA  has  learned  in  manufacturing  settings  that  sustainment 
training  to  criterion  one  GO  was  sufficient  to  raise  the  performance  of 
technicians  to  approximately  100%.  The  ATB  suggested  testing  this  observation 
at  Fort  Lewis.  The  soldiers  were  trained  to  criterion  one  GO,  and  then  given  two 
additional  trials.  Table  9  summarizes  the  percentage  of  soldiers  reaching 
criterion  one  GO  on  the  first  through  the  seventh  trial. 


TABLE  9:  TRIAL  NUMBER  ON  WHICH  SOLDIERS 
MET  CRITERION  ONE  GO 


Trial 

Number 

Percent 

GO 

1 

32.8 

2 

46.2 

3 

16.8 

4 

1.6 

5 

2.0 

6 

0 

7 

0.4 

Ninety-one  percent  of  the  soldiers  scored  two  consecutive  perfect  trials  after 
meeting  criterion  one  GO.  Based  on  this,  we  feel  that  the  soldiers  had  been 
effectively  retrained.  The  soldiers  required  466  sustainment  training  trials  to 
reach  criterion  one  GO.  These  same  soldiers,  learning  these  same  tasks  at  Fort 
Benning,  required  1,659  trials  to  reach  criterion.  Sustainment  training 
represents  a  72%  training  reduction  over  initial  training.  MGA’s  experience  with 
other  training  indicates  that  continued  sustainment  training  reductions  will  be 
demonstrated  over  time.  Practical  application  and  operational  testing  would 
validate  this  for  Brmv  unit  training. 


IV.  TASK  CATEGORIZATION 


BACKGROUND 

ATS  requested  that  as  an  adjunct  to  the  STAR  validation,  this  study  include  a 
test  of  the  ability  to  categorize  tasks  according  to  their  skill  retention 
characteristics.  The  STAR  validation  study  included  the  logistical  requirements 
for  such  a  small  task  categorization  study,  so  it  was  included  as  a  side  issue. 

The  task  categorization  effort  attempted  to  determine  if  tasks  could  be 
categorized  according  to  characteristics  which  relate  to  retention  of  task 
performance.  If  successful,  a  simple  rule  of  thumb  could  be  developed  for 
estimating  refresher  training  cycles  based  on  task  characteristics  alone. 

APPROACH 

MGA  and  ATB  staff  members  selected  a  small  set  of  task  characteristics  which 
are  most  often  related,  according  to  the  research  literature,  to  skill  retention 
and  decay.  Seven  variables  were  selected  to  describe  each  task.  These  variables 
were  scored  as  either  present  or  absent  in  each  task.  Each  variable  received  a 
score  of  1  or  0;  1  if  the  variable  enhanced  retention;  0  if  it  degraded  retention. 
Each  of  the  seven  variables  was  given  equal  weight.  The  seven  variables  were 
summed  for  each  task  providing  a  tac  categorization  score  which  ranged  from  0 
to  7.  The  higher  the  score  the  more  resistant  the  task  should  be  to  skill  decay 
and  the  easier  and  longer  retained. 

These  ‘ask  categorization  scores  were  compared  to  later  retention  performance 
to  determine  if  the  expected  relationship  holds  up.  Because  many  other 
uncontrolled  variables  affect  retention  performance,  a  strong  relationship  is 
necessary  to  make  task  categorization  a  useful  tool.  Previous  attempts  to 
discover  task  features  related  to  retention  and  skill  decay  have  not  yet  yielded 
information  in  a  form  which  can  be  validated  or  readily  used  in  the  field. 

The  results  of  these  studies  appear  to  lack  application  in  the  Army  because: 
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The  relationships  have  not  been  strong  enough  to  warrant  application. 


•  The  retention  results  are  not  estimated  prior  to  training  and  are 
sensitive  to  training  method  and  practice  frequency. 

•  There  is  no  method  for  estimating  proficiency  over  practice  intervals 
that  are  different  from  the  interval  at  which  the  original  retention  data 
were  collected. 

The  task  categorization  effort  of  this  project  was  an  attempt  to  address  these 
issues. 

The  following  seven  task  variables  were  used  for  the  categorization  effort: 

•  Task  position  in  the  POI 

It  is  expected  that  those  tasks  that  are  trained  the  earliest  will  be 
retained  longer.  Also,  these  tasks  are  more  likely  to  be  reinforced 
during  the  training  cycle. 

•  Safety  steps  in  the  task. 

Safety  measures  usually  conflict  with  the  task  mechanics.  Retention  of 
tasks  containing  safety  measures  will  be  degraded. 

Example:  A  DRAGON  gunner  is  required  to  check  the  backblast  area 
prior  to  sighting  and  firing  the  weapon.  This  safety  step  is  quickly 
forgotten. 

•  Cognitive  versus  physical  task. 

Physical  tasks  are  more  likely  to  require  the  integration  of  sensory 


modalities  than  cognitive  tasks.  Predominantly  physical  tasks 
are  retained  longer  than  cognitive  tasks. 

Example:  Firing  the  V160  machinegun  integrates  vision, 

kinesthesia,  hearing,  and  tactile  sensation.  Cognitive  tasks 
require  the  integration  of  mental  skill,  knowledge  and  pro¬ 
cedure  to  produce  the  desired  outcome.  Example  cognitive 
tasks  would  include  SALUTE,  Prepare  an  Antiarmor  Range 
Card,  TOW,  and  Shoot  an  Azimuth. 

•  Number  of  subtasks. 

Breaking  a  successive  string  of  task  elements  into  discrete 
subtasks  decreases  retention. 

Example:  A  sixteen  step  task  taught  as  a  whole  unit  will  be 
retained  better  than  a  sixteen  step  task  that  is  divided  into  two 
or  more  subtasks. 

•  Number  of  steps. 

Short  tasks  are  retained  better  than  long  tasks.  A  long  task  is 
defined  as  17  or  more  steps. 

•  Troubleshooting  or  branching  task  procedures. 

When  task  structure  varies  with  external  conditions  of  the 
"if . .  then  do . "  format,  retention  is  degraded. 

Example:  In  the  task  Treat  a  Blood/Nerve  Agent,  the  soldier 
branches  on  the  basis  of  the  victim's  symptoms.  This  task  is 
quickly  forgotten. 
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Procedural  task. 


• 

A  procedural  task  contains  a  series  of  short,  discrete, 
sequential  motor  responses:  a  response  provides  no  cues  for 
subsequent  responses.  Tasks  having  four  or  more  procedural 
responses  will  not  be  retained  as  well  as  non-procedural  tasks. 

Example:  Performing  Operational  Checks  and  Maintenance 

(PMCS)  on  the  M60  machinegun  requires  memorization  of  a 
series  of  procedures  that  are  not  immediately  related  to  firing 
the  weapon,  and  are  forgotten  more  quickly  than  driving. 

RESULTS 

Pearson  Product  Moment  correlation  coefficients  were  calculated  between  task 
categorization  scores  and  EOC  scores.  Correlation  coefficients  were  calculated 
for  all  tasks  and  for  tasks  within  each  of  the  four  training  methods.  None  of 
these  correlations  was  statistically  significant  nor  numerically  large  enough  for 
practical  utility.  The  finding  that  task  categorization  does  not  relate  to  EOC 
performance  is  not  surprising  since  the  training  is  designed  to  reduce  the 
variability  in  performance  by  the  end  of  course  to  somewhere  between  80%  and 
100%.  This  restriction  in  the  range  of  performance  at  EOC  reduces  the 
likelihood  of  obtaining  a  demonstratable  relationship  between  task  categor¬ 
ization  and  EOC  performance. 

A  more  likely  place  to  find  a  relationship  to  the  task  categorization  variables  is 
in  the  longer  term  retention  performance.  An  index  of  task  retention  was 
calculated  which  indicates  the  amount  of  change  in  task  performance  between 
EOC  and  retention  several  weeks  later.  This  index  is  the  ratio  of  the  in-unit  task 
retention  score  to  EOC  score.  If  this  ratio  is  one  (1),  then  retention  performance 
was  equal  to  EOC  performance  and  there  was  no  skill  decay.  As  the  ratio 
decreases,  retention  performance  decreases  and  the  task  categorization  score 
should  decrease,  if  it,  in  fact,  relates  to  retention  performance.  None  of  the 
correlations  between  task  categorization  scores  ana  the  retention  index  (i.e.,  by 


1 


training  method  and  overall)  were  significant.  Correlations  between  task 
categorization  scores  and  absolute  in-unit  retention  scores  were  also  non¬ 
significant. 

This  lack  of  significant  task  categorization-retention  correlations  may  be  due  to 
one  or  more  of  the  following  reasons: 

•  The  task  variables  used  were  not  appropriate. 


•  Other  factors  besides  the  task  variables  were  stronger  and  overcame 
the  effect  of  the  task  variables. 

•  Tasks  cannot  be  categorized  in  a  simple  way  to  indicate  expected 
retention  performance. 


Further  research  with  more  complex  research  designs  in  task  categorization  may 
produce  methods  and  variables  which  have  the  potential  of  providing  field- 
useable  results.  This  limited  study  did  not  discover  them  through  the  attempt 
described  above. 


DISCUSSION 


This  discussion  will  focus  on  what  was  learned  or  developed  during  this  study  that 
has  application  to  the  Army.  This  study  was  not  basic  research  but  was  an 
operational  verification  of  the  STAR  software.  Overall  verification  of  the  STAR 
software  occurred.  The  software  program  did  provide  useable  EOC  estimates, 
and  in-unit  retention  estimates. 

ESTIMATION  -  CALIBRATION 

The  estimation  standard  employed  for  in-unit  performance  was  minimum  perfor¬ 
mance,  i.e.,  the  lower  boundary  of  performance.  We  estimated  what  the 
minimum  expected  performance  would  be.  This  is  a  cautious  estimate  which 
maximizes  the  assurance  that  the  actual  performance  will  be  equal  to  or  greater 
than  our  estimate.  The  operational  implication  is  that  there  is  massive 
assurance  that  the  skill  will  be  sustained  by  using  the  estimated  interval  for 
sustainment  training.  The  risk  is  that  there  will  be  tasks  in  which  training  is 
given  before  it  is  needed.  As  such,  there  will  be  unnecessary  training  at  an 
additional  cost. 

There  are  several  operational  implications  based  on  the  strategy  selected.  To 
obtain  maximum  assurance  that  a  performance  will  occur,  there  will  be  instances 
of  unnecessary  training  and  unnecessarily  frequent  sustainment  training.  To 
minimize  the  risk  and  associated  cost  of  unnecessary  training  will  result  in  less 
assurance  that  the  desired  performance  will  occur. 

Policy  decisions  ;  npact  directly  on  calibration  procedures  which  provide  EOC 
performance  and  dictate  sustainment  frequency  schedules.  The  percent  of  skill 
decay  to  be  tolerated  before  refresher  or  sustainment  training  dictates  fre¬ 
quency  of  training. 


For  either  HOC  or  sustainment  training,  a  mixed  calibration  approach  can  be 
employed.  Accuracy  or  certainty  of  occurrence  calibration  might  be  higher  for 
critical  or  dangerous  tasks  and  lower  certainty  employed  for  non-critical  tasks. 

It  is  entirely  possible  to  reduce  the  range  of  the  EOC  and  retention  estimates 
and  thereby  decrease  the  training  risk.  The  most  obvious,  probably  the  least 
expensive,  method  is  to  increase  initial  performance  through  evolutionary  change 
in  the  training  materials  and  methods  used  in  the  training  bases.  Volume  II 
contains  an  analysis  of  the  44  tasks  that  were  tested  in  units  and  recom¬ 
mendations  for  training  review.  These  recommendations,  if  implemented,  would 
result  in  improved  EOC  performance  and  reduce  the  range  of  the  retention 
estimates.  Other  broader  changes  in  training  show  potential  promise.  These 
include  adoption  of  a  distributed  (spaced)  training  model  and  the  acceptance  of  a 
higher  training  criterion.  Fort  Benning  has  recently  adopted  a  90%  criterion  for 
POI  qualification  testing.  STAR  simulation  indicates  that  this  criterion  will 
result  in  a  minimum  increase  of  ten  percent  in  EOC  retention  scores.  The 
impact  of  this  new  criterion  can  be  readily  evaluated  with  further  field  testing. 

The  important  points  are:  (a)  the  STAR  software  is  interactive  and  flexible:  (b) 
policy  decisions  dictate  how  the  software  should  be  calibrated  to  provide  cost- 
benefit  trade-offs,  and  (c)  once  calibrated,  the  procedure  is  complete  and  need 
not  be  repeated. 

PERFORMANCE  DATA 

The  STAR  software  requires  a  sample  of  first  trial  data  input  as  a  step-by-step 
performance  measure  for  each  task.  Performance  data  were  also  required  to 
neasure  EOC  performance  and  in-unit  retention. 

The  decision  was  made  to  develop  special  measuring  instruments  for  this  study. 
These  instruments  came  to  be  called  Skill  Sustainment  Exercises  (SSEs).  These 
SSEs  provided  a  consistent  means  of  measuring  task  performance  before  and 
after  training.  Care  was  taken  to  ensure  that  these  performance  measures  were 
based  on  a  front-end  analysis  of  the  tasks,  and  that  the  measures  were  consistent 


with  the  POI.  The  SSEs  are  high  fidelity  reflections  of  tne  requisite  task 
performance.  They  are  consistent  with  but  not  always  identical  to  Soldier 
Manual  Standards.  Both  call  for  the  same  performance  steps  and  require  an 
observable  scorable  action.  The  Soldier  Manual  Standards  also  include  time  and 
condition  standards,  whereas  the  SSE  calls  only  for  demonstrated  ability  to 
perform  the  task. 

The  primary  value  of  SSEs  for  this  study  was  their  utility  as  performance 
measures.  The  procedures  used  in  developing  the  SSEs  ensured  that  they  were 
consistent  the  the  POI  and  the  Soldier  Manual.  Clearly,  if  available,  other 
performance  measures  could  have  been  employed. 

DATA  BASE 

As  a  result  of  this  study,  there  exists  a  major  data  base  for  the  CMF  11.  One- 
hundred-fourteen  tasks  were  extensively  analyzed  and  developed  into  SSEs.  EOC 
estimated  and  actual  performance  for  each  task  is  available.  Estimates  for 
retention  or  sustainment  intervals,  and  verified  data  for  a  subset  of  these  tasks 
exists. 

A  prototype  trainer's  guide  was  developed  from  the  data  base  and  is  attached  as 
an  annex  to  this  report.  This  guide  was  developed  to  demonstrate  a  single 
application  of  the  data  base.  Clearly,  further  work  is  needed  to  implement  data 
base  uses  and  to  integrate  the  data  base  products  into  the  Army. 

The  value  of  such  a  data  base  is  considerable.  EOC  performance  by  task  with 
the  existing  instructional  system  is  known.  If  training  is  revised,  then  the  SSEs 
or  comparable  measures  permit  the  training  developer  to  predict  the  impact  of 
the  change.  Also,  the  sustainment  intervals  by  task  are  known.  Thus,  a 
management  of  a  career  management  field  is  available.  Audits  can  be 
performed,  and  resource  allocation  decisions  checked  out  with  data  of  this  sort. 
These  data  should  be  provided  to  the  Infantry  School  with  the  purpose  of 
becoming  part  of  a  permanent  data  base. 


PROJECT  PRODUCTS 


The  goal  of  this  mutual  effort  between  the  Army  Training  Board  and  MG  A  has 
been  to  operationally  test  the  STAR  technology  to  determine  if  it  can  be 
developed  into  a  practical,  useable,  and  effective  tool  that  can  assist  combat 
unit  commanders  in  attaining  and  sustaining  high  levels  of  soldier  performance. 
The  principal  focus  of  this  work  has  been  to  discover  how  long  it  takes  individual 
soldiers  to  perform  central  tasks  to  an  established  criterion,  to  what  extent  that 
performance  is  degraded  over  time,  and  what  is  required  to  retrain  the  individual 
to  the  desired  level  of  proficiency. 

The  specific  objective  of  this  study  has  been  the  verification  of  the  capability  of 
the  MGA  STAR  software  to  provide  useful  and  practical  data  and  procedures  for 
the  eventual  development  of  such  a  tool.  Overall  verification  of  the  STAR 
software  occurred.  The  results  demonstrate  that  the  STAR  software  can  be  used 
to  estimate  end-of-training  proficiency,  the  length  of  time  such  proficiency  can 
be  maintained  without  retraining,  and  the  training  necessary  to  regain  pro¬ 
ficiency  after  an  interval  of  time.  In  addition,  the  software  has  a  capability  that 
enables  trainers  and  training  developers  to  determine  trade-offs  between  perfor¬ 
mance  outcomes  and  cost  of  training. 

Figure  6  is  a  graphic  recap  of  the  STAR  operational  test  showing  the  products  of 
this  work.  For  a  given  training  delivery  system,  Fort  Benning  POI  in  this  case,  it 
is  possible  to  establish  a  training  CRITERION  (I  in  Figure  6),  that  will  determine 
the  training  effort  required  to  meet  a  desired  level  of  proficiency  at  the  END- 
OF-COUR5E  (II  in  Figure  6).  The  POI  can  be  adjusted  to  achieve  a  desired  level 
of  performance  -  TRAINING  RECOMMENDATIONS  (III  in  Figure  6).  STAR 
retention  estimates  determine  the  frequency  of  necessary  sustainment  training 
on  a  combat  unit-specific  basis,  adjusting  for  individual  unit  training  schedules 
and  priorities  (IV  in  Figure  6).  A  TR  AINING  GUIDE  (V  in  Figure  6)  can  establisn 
training  frequencies  for  tasks  clustered  either  by  weapon  system  retention 
interval  or  collective  training  exercises.  This  training  guide,  including  the 
training  methods  and  criteria,  could  be  used  both  for  sustainment  training  and 
for  initial  unit  training  of  individual  tasks  that  are  not  trained  to  Soldier's 
Manual  Standard  in  the  training  base. 


